Full hardware implementation of neuromorphic visual system based on multimodal optoelectronic resistive memory arrays for versatile image processing

In-sensor and near-sensor computing are becoming the next-generation computing paradigm for high-density and low-power sensory processing. To fulfil a high-density and efficient neuromorphic visual system with fully hierarchical emulation of the retina and visual cortex, emerging multimodal neuromorphic devices for multi-stage processing and a fully hardware-implemented system with versatile image processing functions are still lacking and highly desirable. Here we demonstrate an emerging multimodal-multifunctional resistive random-access memory (RRAM) device array based on modified silk fibroin protein (MSFP), exhibiting both optoelectronic RRAM (ORRAM) mode featured by unique negative and positive photoconductance memory and electrical RRAM (ERRAM) mode featured by analogue resistive switching. A full hardware implementation of the artificial visual system with versatile image processing functions is realised for the first time, including ORRAM mode array for the in-sensor image pre-processing (contrast enhancement, background denoising, feature extraction) and ERRAM mode array for near-sensor high-level image recognition, which hugely improves the integration density, and simply the circuit design and the fabrication and integration complexity.

good flexibility (Supplementary Figures 2d-f), which can be ascribed to the rich hydrogen bonds and hydroxyls that are provided by the Pg-3 and 5, 6-DHI.The MSFP thin film synthesized by the SFP solution, Pg-3 and 5, 6-DHI, showed a uniformly smooth and dense surface (Supplementary Figures 2g-i Supplementary Figures 3b-f show the field emission scanning electron microscope (FE-SEM) images of the SFP sample with different precursor dialyzing times from 12 to 72 hours, showing no obvious change in the SFP surface.After reaction with the Pg-3, the 5, 6-DHI, or both of them, the formed MSFP film surface becomes denser and smoother, showing higher thin film quality (Supplementary Figures 3g-i), which may be responsible for the lower current density and higher stability in the MSFP memory device compared with that in the SFP memory device.The MSFP thin film shows good dielectric properties and strong absorption in the visible light region.The SFP material's dielectric constant (εr) is 8.88 @ 1MHz, consistent with the previous reports 6 .The synthesized MSFP material shows a higher dielectric constant of 40.62@ 1MHz, which exceeds four times than the SFP thin film (Supplementary Figure 5a), enabling the MSFP-based electronic device with good electrical properties such as bias durability, low leakage current, extensibility, and higher charge binding capability 3 .The absorption spectra of MSFP and SFP thin films on the quartz substrate are shown in Supplementary Figure 5b.The MSFP material exhibits stronger absorption in the visible light range than the SFP thin film, enabling the MSFP-based resistive memory with good optoelectronic properties.The multilevel storage of 16 conductance states and retention properties of the MSFP memory device is demonstrated in Supplementary Figure 9.The 16 conductance states are programmed by consecutive electrical pulse groups, with each group composed of five pairs of a programming pulse (0.7V, 50μs) followed by a reading pulse (0.1V, 50μs).All 16 conductance states exhibit nonvolatile properties with a retention time of up to 10 3 seconds, as shown in Supplementary Figure 9.
Supplementary Fig. 9.The 16 nonvolatile conductance states under the 16 electrical pulse groups that each group is composed of ten constant pulses (0.7V, 50μs) followed by reading voltage 0.1V for 1000 seconds.
Supplementary Figure 10a (LTP in Figure 2d) exhibits the conductance potentiation

Supplementary Note 3
The physical mechanism of the analogue resistive switching behaviours.The defectrelated Poisson equation is described as follows 9, 10 : Where the n(), E() represent the charge density and electric field respectively.The  0 ,   , and e are the vacuum permittivity, relative permittivity and electron, respectively.The charge injection is controlled by the external electric field.At a low bias voltage, most of the injected electrons are captured in the defect sites.In this case, the concentration of the free electrons (nf) is negligible.The Nt and nt are defined as the total defects and the electron-filled defects.The nt is given by: Thus, the charge density can be represented as n =  +  .In this case, the Poisson equation can be re-written by: Therefore, the current density can be described by the Mott-Gurrey law 11,12 : where  =  /(  +   ) is the proportion of free electrons.The μ and  denote the mobility and thickness of the switching function layer, respectively.Therefore, the charge density is in the state of   <<  under the low bias voltage.With a relatively high voltage, the conduction is dominated by the Mott-Gurrey law; in other words, the current-voltage relation obeys the I~V m=2 .The current-voltage relation follows the I~Vm>2 in the large bias voltage region.The double logarithm I-V fitting is conducted using the continuous I-V curves (Supplementary Figure 11a).The transition from the low bias voltage region to a relatively high bias voltage region occurs at 0.025 V.The transition from the relatively high bias voltage region to the high bias voltage region occurs at different voltages (0.67, 0.61, 0.57, 0.54 V) for the 1 st , 5 th , 10 th and 15 th cycles, respectively.Therefore, the defect concentration in the MSFP switching layer can be calculated by 10 : Where the   is the transition voltage.According to equation 5, the defect concentrations can be calculated at different bias voltages, respectively (Supplementary Figure 11b).The defect concentration is 0.11×10 16  Comparing the switching between the devices with the dialyzing times of 36 hours and 48 hours, the device with the dialyzing time of 36 hours shows higher current density and a relatively unstable switching effect, while the device with the precursor dialyzing time of 48 hours shows lower current density and better analogue switching behaviours.This may be attributed to that a longer dialyzing time induces less Li + and fewer trap states inside the MSFP thin films.Because of the lower trap concentration, the MSFP memory with a 48-hour precursor dialyzing time shows a lower current density at HRS and more stable switching than that with a 36-hour precursor dialyzing time.Although a relatively high Li + concentration and trap concentration will lead to a higher on/off ratio in the MSFP device with a dialyzing time of 36 hours, the device shows a higher HRS current density and relatively unstable switching.
Supplementary Fig. 12. Switching memory behaviours of the MSFP device with different dialyzing times from 72 to 36 hours.
Since the residual ion, such as Li +, can introduce a certain number of trap sites in the MSFP function layers [1][2][3] , the MSFP memory devices with different Li + concentrations and, thus, different trapping concentrations are fabricated to prove the switching mechanism further.The Li + concentrations can be modulated through the precursor dialyzing times.Therefore, we prepared the MSFP memory devices with different precursor dialyzing times of 36, 48 and 72 hours, respectively, as shown in Supplementary Table 1 and estimated the number of Li + in the MSFP materials by the inductively coupled plasma (ICP) with mass spectrometry, as shown in Supplementary Table 1.The results suggest that the MSFP memory device with a dialyzing time of 72 hours with the least number of Li + and trapping sites shows abrupt switching instead of continuous switching.
In comparison, the devices with dialyzing times of 36 hours and 48 hours and the correspondingly more trapping sites all exhibit continuous and graded switching.The device with the 36-hour dialyzing time shows a higher on/off ratio than the 48-hour dialyzing time because of the higher trap concentration, which also agrees with the proposed switching mechanism.However, the higher trap concentration could also lead to a relatively unstable switching and higher HRS current density.
In summary, the analogue switching behaviours mainly rely on the electron soft filling in the Li + -introduced traps in the MSFP thin film.The device shows a higher on/off ratio when the trap concentration increases.However, the higher trap concentration could also lead to a relatively unstable switching and higher HRS current density.Therefore, the MSFP memory with dialyzing 48 hours was used in this work.First, for the effects of the MSFP thickness on the optical switching behaviours, we studied the optical switching behaviours in the MSFP memory with different MSFP thicknesses of ~63, ~97, and ~186nm, respectively, as shown in Supplementary Figure 15.The MSFP's thickness greatly impacts the resistive switching behaviours, including current density, resistance ratio, and stability.The memory device with a 63 nm MSFP SFP s-c

s-c β-s r-c a-h
function layer shows a relatively high current density of 8~25 A cm -2 at 0.2V and a small resistance ratio (~3) between PPM and NPM states (Supplementary Figure 15a).
A higher resistance ratio (~10) and lower current density (0.35~5 A cm -2 at 0.2V) are obtained in the MSFP memory with ~97 nm MSFP thin film (Supplementary Figure 15b), which may be attributed to the higher secondary structure change volume compared with that of 63 nm.However, the higher resistance states and even smaller resistance ratio (~2) are observed in the MSFP memory with an MSFP thickness of 186 nm, which may be attributed to the fact that the conversion volume of secondary structure in the MSFP thin film is limited under the light illumination with a specific power, therefore the secondary structure conversion ratio and the resistance change ratio is relatively small when the MSFP thin film thickness is relatively thick (186 nm), and the as-prepared MSFP memory device is already at very HRS (Supplementary Figure 15c).
For the area influences on the conductance, the optical switching behaviours of MSFP memory devices with different device areas of 100×100μm 2 , 150×150μm 2 and 200×200 μm 2 are studied as shown in Supplementary Figure 15d.It can be noted that the memory device with a large area exhibits a wider conductance range for both the NPM and PPM effects.The memory cell with an area of 200×200μm 2 shows an optical switching ratio of ~10, while this ratio respectively decreases to ~8 and ~5 when the area decreases to 150×150μm 2 and 100×100μm 2 , as shown in Supplementary Figure 15d.The photoconductance change ratio shows increasing trends with the increased device area since the large area corresponds to a higher volume of the secondary structure change, thus a larger photoconductance change.The area-dependent switching is also consistent with the conductance change in the MSFP thin films that arises from the secondary structure change.
When changing the device area and the MSFP thin film thickness, the critical intensity is nearly unchanged.The crucial intensity in the change from PPM to NPM effect mainly depends on the heat provided by the light [13][14][15] .Therefore, the threshold To investigate the impact of the electrical switching on the optical switching, the Au/MSFP/Au resistive memory is first set to the low resistance state (LRS) by 100 unaffected optical switching behaviours (Supplementary Figure 16).Therefore, the electrical switching nearly does not impact the optical switching behaviours.

). Supplementary Fig. 2 .
Synthesis of MSFP switching thin films.a. Natural silk from the cocoon LG-2.b.Fibroin extracted from cocoon LG-2 and precursor solution of the MSFP.c. Preparation of the flexible MSFP substrate after thermal processing.d-f.Optic images of the synthesized MSFP substrate.g-i.FE-SEM images of the MSFP film spin-coated by the MSFP precursor solution.The film presents a dense and smooth state.The XRD and SEM characterized the changes in crystallographic and structural properties of SFP and MSFP thin films.Supplementary Figure 3a exhibits the XRD patterns of the SFP thin film prepared under different conditions, including the precursor dialyzing time of 12, 24, 36, 48 and 72 hours, a mixture of Pg-3 and SFP with the dialyzing time of 48 hours, a mixture of 5,6-DHI and SFP the dialyzing time of 48 hours, and the MSFP thin film with the SFP precursor dialyzing of 48 hours.No noticeable crystallographic changes are detected between SFP and MSFP thin films.

Supplementary Fig. 3 .
Crystallographic and structural characterizations of SFP and MSFP thin films.a.The XRD pattern of the SFP thin film with various precursor dialyzing times from 12 to 72 hours and the MSFP thin film formed by the reaction of Pg-3, 5,6 DHI with SFP.b-f.FE-SEM images

Supplementary Fig. 4 .
thin films with dialyzing times of12, 24, 36, 48, and 72 hours, respectively.g-j.FE-SEM image of the SFP thin films and MSFP thin film (dialyzing time of 48 hours), respectively.The scale bar represents 500 nm.The FTIR results between SFP and MSFP demonstrated that the strong chemical bond vibration of the -OH, -C=O, C=C, and C-OH respectively located at 3278, 1635, 1520 and 1230 cm -1 are observed in both SFP and MSFP thin films while a weak -CH2(s) vibration located at 2856 cm -1 is observed in the MSFP thin film, indicating that the major structures of the SFP can be reserved in the MSFP after the chemical reaction between SFP, 5,6-DHI and Pg-3 and the structures such as the connection between protein chains in the MSFP are changed during the reaction.In addition, the chemical bond of C-O located at 1165 cm -1 in the SFP is not detected in the MSFP (Supplementary Figure4), indicating that the C-O-based bond groups can possibly provide the chemical reaction sites to form a large number of hydrogen bonds among USFP, Pg-3 and 5,6-DHI[5][6][7][8] .FTIR spectra of the SFP and MSFP.The primary structures of SFP were reserved in the MSFP thin film; however, the -CH2-based chains were generated in the MSFP thin film.

SupplementaryFig. 8 .
by abrupt SET and RESET process, which may be attributed to the formation of the Li + -based conduction paths.Memory switching behaviours of the SFP device with different dialyzing times from 72 to 36 hours.

Supplementary Fig. 15 .
heat and optical power for turning PPM to NPM are nearly identical for devices with different areas or thicknesses.The influences of device area and MSFP thickness on the optical switching behaviours.a-c, The optical switching behaviours in the MSFP memory devices with different MSFP thicknesses of ~63, ~97, and ~186nm, respectively.d, Device area dependent optical LTPs and LTDs in the memory cell with different device areas of 100×100, 150×150 and 200×200μm 2 .
Au/~97 nm MSFP/Au electrical pulses (0.7V, 50μs).Then, at this LRS triggered by the electrical pulses, we further employ 100 PPM optical pulses (80mW, 200ms) to set the device to a lower LRS, and then this lower LRS can be reset back to the low resistance state (LRS) through 100 NPM optical pulses (40mW, 200ms).The LRS is reset to its initial HRS baseline through 100 electrical pulses (-0.7V, 50μs).After the above operation, the memory device can still be set and reset by the light pulses (set by 50 optical pulses (80mW, 200ms) and then reset by 50 optical pulses (40mW, 200ms)), showing

Table 4 .
Comparison of the optoelectronic memory devices based on different photoelectronic function materials.